An information-theoretic derivation of min-cut based clustering 
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Min-cut clustering, based on minimizing one of two heuristic cost-functions proposed by Shi and 
Malik, has spawned tremendous research, both analytic and algorithmic, in the graph partitioning 
and image segmentation communities over the last decade. It is however unclear if these heuristics 
can be derived from a more general principle facilitating generalization to new problem settings. 
Motivated by an existing graph partitioning framework, we derive relationships between optimizing 
relevance information, as defined in the Information Bottleneck method, and the regularized cut in 
a K-partitioned graph. For fast mixing graphs, we show that the cost functions introduced by Shi 
and Malik can be well approximated as the rate of loss of predictive information about the location 
of random walkers on the graph. For graphs generated from a stochastic algorithm designed to 
model community structure, the optimal information theoretic partition and the optimal min-cut 
partition are shown to be the same with high probability. 
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1. INTRODUCTION 

Min-cut based graph partitioning has been used suc- 
cessfully to find clusters in networks, with applications in 
image segmentation as well as clustering biological and 
sociological networks. The central idea is to develop fast 
and efficient algorithms that optimally cut the edges be- 
tween graph nodes, resulting in a separation of graph 
nodes into clusters. Particularly, since Shi and Malik 
successfully showed [T] that the average cut and the nor- 
malized cut (defined below) were useful heuristics to be 
optimized, there has been tremendous research in con- 
structing the best normalizcd-cut-based cost function in 
the image segmentation community. 

The Information Bottleneck (IB) method [21 [3] is a 
clustering technique, based on rate-distortion theory [J], 
that has been successfully applied in a wide variety of 
contexts including clustering word documents and gene- 
expression profiles [5] . The IB method is also capable of 
learning clusters in graphs and has been used successfully 
for synthetic and actual networks In the hard clus- 
tering case, given the diffusive probability distribution 
over a graph, IB optimally assigns probability distribu- 
tions, associated with nodes, into distinct groups. These 
assignment rules define a separation of the graph nodes 
into clusters. 

We here illustrate how minimizing the two cut-based 
heuristics introduced by Shi and Malik can be well- 
approximated by the rate of loss of relevance information, 



* Electronic address: ar2384@columbia.edu 



defined in the IB method applied to clustering graphs. To 
establish these relations, we must first define the graphs 
to be partitioned; we assume hard-clustering and the 
cluster cardinality to be K. We show, numerically, that 
maximizing mutual information and minimizing regular- 
ized cut amount to the same partition with high probabil- 
ity, for more modular 32-node graphs, where modularity 
is defined by the probability of inter-cluster edge con- 
nections in the Stochastic Block Model for graphs (See 
Numerical Experiments! ■ We also show that the op- 



timization goal of maximizing relevance information is 
equivalent to minimizing the regularized cut for 16-node 
graphs. [T2"] 



2. THE MIN-CUT PROBLEM 

Following [7 , for an undirected, unweighted graph Q — 
(V,E) with n nodes and m edges, represented [13] by 



its adjacency matrix A := {A 



■<•.</ 



1 ^> x ~ y}, 
we define for two not necessarily disjoint sets of nodes 
V + ,V_ C V, the association 



W(V + ,V_) = Yl 



A 



xy ■ 



(2.1) 



iev + ,jev_ 
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We define a bisection of V into ~V± if V + U V_ = V 
and V + n V_ = 0. For a bisection of V into V + and 
V_, the 'cut' is defined as c(V+,V_) = T^(V+,V_). 
We also quantify the size of a set V + C V in terms of 
the number of nodes in the set V + or the number of edges 



2 



with at least one node in the set V + : 

W (v+) = 1 

iev + 

fi ( v +) = E rf *> 



(2.2) 



lev. 



where is the degree of node x. 

Shi and Malik [1 defined a pair of regularized cuts, for 
a bisection of V into V + and V_ ; the average cut was 
defined as 



A = 



t^(v+,v_) w{v + .y_) 



uj(V+) w(V_) 
and the normalized cut was defined as 

W(V + ,V_) , TF(V+,V_) 



J\f = 



fi(V+) 



fi(V_) 



(2.3) 



(2.4) 



This definition can be generalized, for a X-partition of 
Vinto Vi,V 2 ,...,V K 71, to 



E 



u,(V,-) 
V ^(V 3 ,Vj) 



(2.5) 
(2.6) 
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where Vj = V \ Vj. 

For the graph 5, we can define the graph Laplacian 
A = D A where D is a diagonal matrix of vertex 
degrees. For a bisection of V, we also define the partition 
indicator vector h [5] 



h- = 



-l Vxev+ 

-1 Vx € V_. 



(2.7) 



Specifying two 'prior' probability distributions over the 
set of nodes V : (i) p{x) oc 1 and (ii) p{x) oc d x , we then 
define the average of h to be 



(h) = 



X^gv d x h x 
2m 



(2.8) 



The cut, as defined by Fiedler [8], and the regularized 
cuts, as defined by Shi and Malik [T], can then by written 
in terms of h as (See |Appendix ) 



A = 



ih T Ah 

4 

1 h T Ah 



n 1 - h 
1 h T Ah 

2^ 1 - (h) 2 



(2.9) 



More generally, for a if-partition, we define the parti- 
tion indicator matrix Q as 



Q zx = p(z\x) = 1 Vx E z 



(2.10) 



where z G {Vi, V2, ~V k } and define P as a diago- 
nal matrix of the 'prior' probability distribution over the 
nodes. The regularized cut can then be generalized as 



c = E 



[Q T AQ]j 



(2.11) 



where for p{x) oc 1, C = A; and for p{x) oc d x , C = Af . 

Inferring the optimal h (or Q), however, has been 
shown to be an NP-hard combinatorial optimization 
problem [3]. 



3. INFORMATION BOTTLENECK 

Rate-distortion theory, which provides the foundations 
for lossy data compression, formulates clustering in terms 
of a compression problem; it determines the code with 
minimum average length such that information can be 
transmitted without exceeding some specified distortion. 
Here, the model-complexity, or rate, is measured by the 
mutual information between the data and their represen- 
tative codewords (average number of bits used to store a 
data point). Simpler models correspond to smaller rates 
but they typically suffer from relatively high distortion. 
The distortion measure, which can be identified with loss 
functions, usually depends on the problem; in the sim- 
plest of cases, it is the variance of the difference between 
data and their representatives. 

The Information Bottleneck (IB) method [3] proposes 
the use of mutual information as a natural distortion 
measure. In this method, the data are compressed into 
clusters while maximizing the amount of information that 
the 'cluster representation' preserves about some speci- 
fied relevance variable. For example, in clustering word 
documents, one could use the 'topic' of a document as 
the relevance variable. 

For a graph Q, let X be a random variable over graph 
nodes, Y be the relevance variable and Z be the random 
variable over clusters. Graph partitioning using the IB 
method [6] learns a probabilistic cluster assignment func- 
tion p(z\x) which gives the probability that a given node 
x belongs to cluster z. The optimal p{z\x) minimizes the 
mutual information between X and Z , while minimizing 
the loss of predictive information between Z and Y . This 
complexity-fidelity trade-off can be expressed in terms of 
a functional to be minimized 



T[p{z\x)] = -I[Y-Z]+TI [X;Z] 



(3.1) 



where the temperature T parameterizes the relative im- 
portance of precision over complexity. As T — > 0, we 
reach the 'hard clustering' limit where each node is as- 
signed with unit probability to one cluster (i.e p(z\x) G 

{0,1}). 
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Graph clustering, as formulated in terms of the IB 
method, requires a joint distribution p{y, x) to be defined 
on the graph; we use the distribution given by continuous 
graph diffusion as it naturally captures topological infor- 
mation about the network [B]. The relevance variable Y 
then ranges over the nodes of the graph and is defined as 
the node at which a random walker ends at time t if the 
random walker starts at node x at time 0. For continu- 
ous time diffusion, the conditional distribution p t (y\x) is 
given as 



imated as: 



G*=p\y\x)=e 



tAP 



(3.2) 



where A is the graph Laplacian and P a diagonal ma- 
trix of the prior distribution over the graph nodes, as 
described earlier. The characteristic diffusion time scale 
t of the system is given by the inverse of the smallest 
non-zero eigenvalue of the diffusion operator exponent 
AP _1 and characterizes the slowest decaying mode in 
the system. To calculate the joint distribution p(y, x) 
from the conditional G*, we must specify an initial or 
prior distribution |14j ; we use the two different priors 
p(x), used earlier to calculate the expected value of h 
: (i) p(x) cx 1 and (ii) p{x) oc d x . 



I [Y, Z] ( In p ^ p ^ J 

'{(l + r ) )]n(l + r ] )) y ] 



I 1 

V + 



\^p{v)p{z) 



(4.2) 



p{z,y) 

p(z)p(y) 



- 1 




p(y,z) 2 

p(y)p(z) 



(4.3) 



Here, we define i as a first-order approximation to / [Y; Z] 
in the well-mixed limit of graph diffusion. 



1. Well-mixed K -partitioned graph 



RATE OF INFORMATION LOSS IN GRAPH 
DIFFUSION 



We analyze here the rate of loss of predictive infor- 
mation between the relevance variable Y and the cluster 
variable Z, during diffusion on a graph Q, after the graph 
nodes have been hard-partitioned into K clusters. 



A. Well-mixed limit of graph diffusion 



For a given partition Q of the graph, defined in Eqn. 
(2.101, we approximate the mutual information I [Y; Z] 



when diffusion on the graph reaches its well-mixed limit. 
We introduce the dependence rj(y, z) such that 



p(y,z) =p(y)p(z)0- + v)- 



(4.1) 



This implies (ij) y = (rj) z = and ((v 2 ) z ) y = (v) where 
() denotes expectation over the joint distribution and 
() and Q z denote expectation over the corresponding 
marginals. 

In the well- mixed limit, we have r\ <C 1. The predic- 
tive information (expressed in nats) can then be approx- 



As in the IB method, the Markov condition Z — X — Y 
allows us to make several simplifications for the condi- 
tional distributions and associated information theoretic 
measures. For a i^-partition Q of the graph, we have 

p{y,z) = z^p{x,y,z) 

X 

= ^2p{z\y,x)p(y\x)p(x) 

X 

= J2P( Z \ X )P(V\ X )P( X ) = QPG tT (4.4) 

X 

p(y,z) 2 = ^J^p(z|a;)p(y|a;)p(a;)^ 

n 

= X] p( z \ x )p(y\ x )p( x )p( z \ x/ )p(y\ x/ )p( x ') 

x,x' — l 

Tl 

= ^2 QzxGlxPxQzx'Glx'Px'- (4.5) 
x,x' — l 

p(z) = ^2p(z\x)p(x) 

X 



Graph diffusion being a Markov process, we have 
E^=i ^i'h^vi = ^x'x- Using this and Baycs rule 
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G\ X P X = Gl y Py, we have 



We then have 



1 / ^-^ Ei,i'=i QzxG yx P x Qzx'G yx iP„ 



E 

1 / Ei,i'=l QzxQzx' PyG x i y G yx P x 

2 ^Z x nQzx"Px")Py 



2\ / —' (Ex" Qzx"Px")Py 

it nt 



- 1 



1 / Ex,x' = l QzxQzx'{Y.y=l G x ,yG yx )P x \ 

1 | Ex,x'=l Q zxQ zx> G x l , X P X \ 



In the hard clustering case, Ex QzxPx = p(z) 
[QPQ T ] ZZ and we have 



z{h [QPQ T ]zz 1 J • (4 - 8) 



2. Well-mixed 2-partitioned graph 

We can re-write t as 
1 



2 V" >y _ 
1 { { {p{z\y)-p{z)f 
2\\ 



(4.9) 



zl y 



For a bisection h of the graph, z e {+1,-1} and we have 
p(z\x) = l -{l±h x )= l -{l + zh x ). (4.10) 



= ^ E(! + z/l *)p( x ^) 

= 1(1 + *<%». (4.11) 
= ^p(z,a;) = ^2p(z\x)p(x) 

X X 

= \ E^ 1 + zh *)p( x ) 



2(1 + ^)). 



(4.12) 



p(z\y)-p(z) = i(l + z(h|y))-i(l + z(h)) 



= -*«%) - (h)). 



(4.13) 



(p(z\y) -p{z)f 
p{zf 



« l((h\y) (h)) 2 

((%) - (h)) 2 * 1 

2 ^ 1 + z(h) 



(<%) - W) 2 
1 - (h) 2 



(4.14) 



The mutual information I [Y; Z] can then be approxi- 
mated as 

! <«%) - (h)) 2 )^ 



2 1 - (h) 2 



2 1 - (h) 2 ' 

Using Bayes rule p t (x\y)p(y) = p t (y\x)p(x), we have 



(4.15) 



<%) = 5>*P*(sfo) = £ 

CC X 

n n 

<(h|y) 2 ) y = J] ^ 



ftxP*(y|a;)p(a;) 



.(4.16) 



-1 x,x' — l 

n n 



p(y) 

p t (y\x)p(x)p t (x'\y) 

p{y) 



= E E '»xW(l/ky(x , |y)p(a:).(4.17) 

y=l x,x'=l 

Again, graph diffusion being a Markov process, 



((hly) 2 )^ = Y, Khxip 2t {x'\x)p{x) 

x.x' — l 

= {h x h x ')2t- 
a 2 «hk,)) = ((h| y ) 2 ) a -(h) 2 

= (h x h x ') 2 t - (h) 2 . 
1 (Wt - (h) 2 
L 2 l-(h) 2 



(4.18) 

(4.19) 
(4.20) 



B. Fast-mixing graphs 

When diffusion on a graph reaches its well-mixed limit 
in short times, we have G 2t wl - 2f AP -1 . Thus, for a 
if-partition of a graph 

Q(G 2 *P)Q T ss Q(P 2tA)Q T 

= QPQ T -2tQAQ T . (4.21) 
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For bisections, the short-time approximation of (h x h x >)2t 
can be written as 



x,x' — l 

h T G 2t Ph 



h x >p 2t (x',x)h a 



h T (I- 2tAP" 1 )Ph 



h T Ph 



2ih T Ah 



1 



2ih T Ah. 



(4.22) 



For fast-mixing graphs, the long-time and short-time ap- 
proximations for / [Y; Z] and (h x h x /}2t, respectively, hold 
simult aneously. 



I[Y;Z] 

dI[Y;Z] 
dt 



t 



dl 



h T Ah 

l-{\v)\ 

p(x) oc 1 
TV ; p{x) oc d x 



A 



(4.23) 



We have shown analytically that, for fast mixing 
graphs, the heuristics introduced by Shi and Malik are 
proportional to the rate of loss of relevance information. 
The error incurred in the approximations I [Y; Z] w t 



and (h x h x ,) 2 t - 
£ 1 {t) 



1 -2fh A Ah can be defined as 



(h x h x ,) 2t - (1 - 2ih T Ah) 



(h x h x /) 2 t 
I[Y;Z] (t)-L(t) 



I[Y;Z] (t) 



(4.24) 
(4.25) 



5. NUMERICAL EXPERIMENTS 

The validity of the two approximations can be seen in 
a typical plot of £\{t) and £o(t) as a function of normal- 
ized diffusion time t = t/r, for the two different choices of 
prior distributions over the nodes. £±, as seen in Fig. [T] 
is often found to be non-monotonic and sometimes ex- 
hibits oscillations. This suggests defining Eoa, a modified 
monotonic t £\: 



Jt) = max£i(t'). 

t'>t 



(5.1) 



We don't need to define a monotonic form for £ since 
this error is always found to be monotonically increasing 
in time. 

By fast-mixing graphs, we mean graphs which become 
well-mixed in short times, i.e. graphs for which both the 
long-time and short-time approximations hold simulta- 
neously within a certain range of time t*_ < t < as 



illustrated in Fig. [T] where we define 

£(t) = max(f oo (<),£ (t)) 

£* = min£(t) 
t 

t*_ = min(argmin£(t)) 



tl = 



max(arg min £{t) ) . 
t 



(5.2) 
(5.3) 

(5.4) 
(5.5) 




0.5 1 
Normalized Time (t/T) 



FIG. 1: £ i and £o vs normalized diffusion time for two choices 
of priors over the graph nodes. £\ (red) typically tends to 
have a non-monotonic behavior which motivates defining a 
monotonic £oo (green). 



Note that the use of £ ^ instead of £\ over-estimates the 
value of £*; the £*'s calculated is an upper bound. 

Graphs were drawn randomly from a Stochastic Block 
Model (SBM) distribution [10], with block cardinality 2, 
to analyze the distribution of £ *, t*_ and t* + . As is com- 
monly done in community detection [11] , for a graph of 
n nodes, the average degree per node is fixed at n/4 for 
graphs drawn from the SBM distribution: two nodes are 
connected with probability p + if they belong to the same 
block, but with probability p_ < p + , if they belong to 
different blocks. The two probabilities are, thus, con- 
strained by the relation 



*(=-•) MS) 



n 
4 



(5.6) 



leaving only one free parameter p_ that tunes the 'mod- 
ularity' of graphs in the distribution. Starting with a 
graph drawn from a distribution specified by a p_ value 
and specifying an initial cluster assignment as given by 
the SBM distribution, we make local moves — adding or 
deleting an edge in the graph and/or reassigning a node's 
cluster label — and search exhaustively over this move- 
set for local minima of £ * . Fig. [2] compares the values of 
£* and {t*_,t* + } for graphs obtained in this systematic 
search, starting with a graph drawn from a distribution 
with p_ = 0.02 and n = {16,32,64}. We note that 
the scatter plots for graphs of different sizes collapse on 
one another when £* is plotted against normalized time, 
confirming the Fiedler value 1/r to be an appropriate 
characteristic diffusion time-scale as used in [B]. A plot 
of £* against actual diffusion time shows that the scatter 
plots of graphs of different sizes no longer collapse 

Having shown analytically that for fast mixing graphs, 
the regularized mincut is approximately the rate of loss of 
relevance information, it would be instructive to compare 
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FIG. 2: £* vs t* for graphs of different sizes and different prior distributions over the graph nodes. In the above plot, t*_ and 
t* + are represented by • and o, respectively. 
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FIG. 3: p(hi n f(t) h cut ) vs normalized diffusion time, aver- 
aged over 500 graphs drawn from a distribution parameterized 
by a given p_ value, is plotted for different graph distributions 



the actual partitions that optimize these goals. Graphs of 
size n = 32 were drawn from the SBM distribution with 
p_ = {0.1,0.12,0.14,0.16}. Starting with an equal-sized 
partition specified by the model itself, we performed iter- 
ative coordinate descent to search (independently) for the 
partition that minimized the regularized cut (h cut ) and 
one that minimized the relevance information (h[ n f(t)); 
i.e. we reassigned each node's cluster label and searched 
for the reassignment that gave the new lowest value for 
the cost function being optimized. Plots comparing the 
partitions h; n f(i) and h cut , learnt by optimizing the two 
goals (averaged over 500 graphs drawn from each distri- 
bution), are shown in Fig. [3] 



6. CONCLUDING REMARKS 

We have shown that the normalized cut and average 
cut, introduced by Shi and Malik as useful heuristics to 
be minimized when partitioning graphs, are well approx- 
imated by the rate of loss of predictive information for 
fast-mixing graphs. Deriving these cut-based cost func- 
tions from rate-distortion theory gives them a more prin- 
cipled setting, makes them interpretable, and facilitates 
generalization to appropriate cut-based cost functions in 
new problem settings. We have also shown (see Fig. [2]) 
that the inverse Fiedler value is an appropriate normal- 
ization for diffusion time, justifying its use in [6] to cap- 
ture long-time behaviors on the network. 

Absent from this manuscript is a discussion of how 
not to overpartition a graph, i.e. a criterion for selecting 
K. It is hoped that by showing how these heuristics can 
be derived from a more general problem setting, lessons 
learnt by investigating stability, cross-validation or other 
approaches may benefit those using min-cut based ap- 
proaches as well. Similarily, by showing how these heuris- 
tics approximate costs functions from a separate opti- 
mization problem, it is hoped that algorithms employed 
for rate distortion theory, e.g. Blahut Arimoto, maybe 
be brought to bear on min-cut minimization. 



APPENDIX 

Using the definition of A, for any general vector f over 
the graph nodes, we have 



f T Af = f T Df-f T Af 



E 



fxfyA xy 

x,y=l 
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I J /a; fxfyA xy 

x \y=l I x,y=l 



_^ / n n n \ 

= 2 ( E fx^-xy — 2 ^ fxfyA X y + ^ fyA xy J 
\a:,y— 1 X ,y—1 X ,y—1 / 



(A.l) 



Now, when f = h, we have 



h T Ah = ; ^ ^ 



h x xh y = — l 
4 X C. 



(A.2) 



The factor ^ disappears because summation over all 
nodes counts each adjacent pair of nodes twice. 
Using the definitions of A and Af, we have 



/ 

A = c x 

- cx | 

= 2c x 

= 2c x 

= 2c x 
4 c 



\ 



, E 1 Ei 

\h x =+l h x =-l 



" l-h x 



Ex ( 2 ) V 2 

E a (i-fex + i + M 

2n 

( n + E a; ^)(""E 2: M 
2 



n(l + h)(l-h) 



ni-h 



2 • 



(A.3) 



N = 



c x 



C X 



. E d - E d - , 



= 2c x 
= 2c x 
= 2c x 



J2x d xQ- -h x + l + h x ) 

E, + (rfx(i-M) 

4m 

(2m + Ez h x d x )(2m - E x 
1 

m(l + (h))(l-(h)) 



ml -(h) 2 ' 



(A.4) 
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